🗒️ Description

Introduces a new CLI tool benchmark_coverage that generates a markdown/JSON/terminal report showing which opcodes have benchmark test coverage for different benchmark modes (--gas-benchmark-values & --fixed-opcode-count mode).

• Scans benchmark tests and generates coverage report for each opcode
  • Distinguishes between two benchmark modes:
    • worst-case-benchmark (--gas-benchmark-values): Tests using benchmark_test wrapper
    • fixed-opcode-count (--fixed-opcode-count): Tests using benchmark_test + target_opcode= + code_generator=
  • Imports opcode definitions from ethereum.forks.osaka.vm.instructions.Ops for accurate sorting and completeness
  • Handles opcode aliases (SHA3 -> KECCAK256, DIFFICULTY -> PREVRANDAO).
  • Outputs markdown table sorted by opcode hex value

Detection logic:

Usage:

  uv run benchmark_coverage           # Generate markdown report
  uv run benchmark_coverage --json    # Output as JSON
  uv run benchmark_coverage -o out.md # Write to file

Current Coverage Report

🔗 Related Issues or PRs

Require #1869

✅ Checklist

• All: Ran fast tox checks to avoid unnecessary CI fails, see also Code Standards and Enabling Pre-commit Checks:

  uvx tox -e static

• All: PR title adheres to the repo standard - it will be used as the squash commit message and should start type(scope):.
• All: Considered adding an entry to CHANGELOG.md.
• All: Considered updating the online docs in the ./docs/ directory.
• All: Set appropriate labels for the changes (only maintainers can apply labels).
• Tests: Ran mkdocs serve locally and verified the auto-generated docs for new tests in the Test Case Reference are correctly formatted.
• Tests: For PRs implementing a missed test case, update the post-mortem document to add an entry the list.
• Ported Tests: All converted JSON/YML tests from ethereum/tests or tests/static have been assigned @ported_from marker.

Cute Animal Picture